Certain physiological measures, such as heart rate and respiratory rate, can provide insight into a person's emotional state. For example, that a person has an elevated heart rate can suggest that the person is excited or distressed. Thus, characteristics of physiological measures can act as physiological cues indicating a person's emotional state.
A person's heart rate and respiratory rate can be determined in contactless fashion using a camera and a computing device. For instance, a person's heart rate can be determined by taking advantage of the blood pulsation effect—small fluctuations in average skin color intensity that occur as blood passes through capillaries close to the skin surface as the person's heart beats. The blood pulsation effect exists because of how light interacts with human skin. Ambient light is reflected or absorbed by human skin in varying amounts due to the presence of three pigments—melanin, carotene and hemoglobin. The amounts of melanin and carotene in skin typically do not vary over the course of a typical sampling timeframe, but the amount of hemoglobin does varies with the action of blood pulsing into and out of the skin, creating a detectable wave in the average color output of a captured area of the face.
These skin color fluctuations are minute and have been measured at about +/−1% of the average unmodulated skin color intensity. The skin color fluctuations are present in all color channels of a video recording of a person and in the red-green-blue (RGB) color space, the effect is more pronounced in the green channel. In the YCbCr color space (a luminance-chroma color space), the effect is most pronounced in the Y (luminance) channel, with a matching lower-amplitude fluctuation in the Cr channel, one of the chroma channels.
A person's respiratory rate can be determined by detecting the rate at which a person's shoulders and/or chest rises and falls in a video.